Open sourcing AI app development with Harrison Chase from LangChain

0:05Hi, listeners, and welcome to another episode of NoPriors. Today, we're talking to Harrison Chase, the CEO and co-founder of LangChain, a popular open source framework and developer toolkit that helps people build LLM applications. We're excited to talk to Harrison about the state of AI application development, the open source ecosystem, and its open questions. Welcome, Harrison. Thanks for having me. I'm excited to be here. LangChain is a really unique story, and it started actually as a personal project for you. Can you talk a little bit about what what Langchain is and what it was originally?

0:37Yeah, absolutely. So how I would answer the question what Langchain is has kind of evolved over time, as has the entire landscape. Langchain, the open source package, started, yeah, as a side project. So my background's in ML and MLOps. I was at my previous company. I knew I was going to leave. I didn't know what I was going to do. This was in September, October of 2022. And so I went to a bunch of hackathons, went a bunch of meetups, chatted with folks that were playing around with LLMs and saw some common abstractions, put it in a Python project as a just fun side project. Turned out to strike a chord, be fantastic timing.

1:15You know, ChatGPT came out like a month later and it's kind of evolved from there. So right now, LangChain, the company, there's really two main products that we have. One is the LangChain open source packages. I'm happy to dive into that more. And then the other There is LangSmith platform for testing, evaluation, monitoring, and all of those types of things. And so, you know, what LangChain is has evolved over time as the company's grown. One thing that we talked about the last time we saw each other in person was just how quickly like the AI ecosystem and research field is evolving and what it means to manage an open source project through that.

1:56Can you talk a little bit about what you decide to keep stable and change when you both have like big ecosystem of users now and like very rapidly changing environment of applications and technology? That's been a fun exercise. So, I mean, if we go back to the original version of LinkChain, what it was when it came out was essentially three kind of like high level implementations. Two were based on research papers and then one was based on Nat Friedman's like NatBot type of agent web crawler thing. And so there was some high level kind of like abstractions. And then there was a few like integrations.

2:28So we had integrations with, I think, like OpenAI, Cohere, and Hugging Face to start or something like that. And those two layers have kind of like remained. So we have, you know, 700 different integrations. We have a bunch of kind of like higher level chains and agents for doing particular things. I think the thing that we've put a lot of emphasis in, to your point around kind of like what's remained constant and what's changed, is like a lower level kind of like abstraction and runtime for joining these things together. One of the things that we pretty quickly saw was that as people wanted to improve the performance, go from prototype to production, they wanted to customize a lot of these bits.

3:04And so we've invested a lot in a lower level kind of like chaining protocol. So Lange chain expression language. And then in a different protocol, Lange graph, which is something we're really excited about. And that's more aimed at basically graphs that are not DAGs. So, you know, all these agents are basically running an LLM in a loop. You need cycles. And so Lange graph helps with that. And so I think what we've kind of seen is the underlying bits of there's all these different integrations. And like, you know, there's there's LLMs, vector stores, and sometimes they change, right? When chat models came out, like that was it.

3:41That's a very big change in the API interface. And so we had to add a new abstraction for that. But those have, especially over the past few months, remained relatively stable. We've invested a lot in this underlying runtime, which emphasizes a few things, streaming, structured outputs, and the importance of those has remained relatively stable. But then the way that you put things together and the kind of like patterns for building things has definitely evolved over time from like simple chains to complex chains to then these kind of like autonomous agents to now something maybe in the middle of like complex state machines or graphs or something.

4:18And so it's really that upper layer, which is like the common ways to put things together that I think we've seen the most rapid kind of like churn. What do you think is still missing from really getting to performant agents? There's a number of companies that have been started recently that are really focused on sort of the agentic world and pushing that whole thread in certain types of automation forward. What do you view as the big components that you all don't have or that maybe the industry more generally doesn't have that still needs to come into place to help drive those things ahead? Yeah, that's a really good question.

4:45I think there's a few things. One, I think like figuring out the right UX for a lot of these things is still an open question in my mind. And, you know, that's not necessarily something we can help with. I think there's a lot of exploration that applications need to do to figure out how to, you know, communicate what these agents are good at and bad at to end users and expose ways to maybe let them course correct and see what's going on. And so, you know, I think we try to emphasize a lot of this observability of intermediate steps and even correcting intermediate steps. But there's a lot of experimentation around UX that I think needs to happen.

5:20Um, another big part I think is, is basically the planning ability of the underlying LLMs. Um, I think that's probably the biggest, I think when we see people building agents that work right now, it's often breaking it down into a bunch of smaller components and, and kind of like imparting their domain knowledge about how information should flow through these components. Um, because I think the LLMs by themselves still aren't able to, to reason fully about how that should happen. And I think we see a few kind of like, a lot of research is actually around this, I would say in the academic space.

5:57Specifically, I think there are two different types of research papers around agents that we see. We see some around like planning for agents. So there's a bunch of papers that do kind of like an explicit planning step up front. And then there are other research papers that do a bunch around reflection. So like after it, after an agent does something, is this actually right? How can I kind of like, you know, improve upon that? And I think both of those are basically trying to get around the shortcomings of LLMs, and that in theory, they should do that automatically, right? Like you shouldn't have to ask an LLM to plan or to think about whether what it's done is correct, it should know to do that.

6:35And then it can kind of like run in a cycle. But we see a lot of shortcomings there. And so I think planning the ability of LLMs is a big one. And that'll get better over time. The last one is maybe a little bit more vague, but I think even just as builders, we're still figuring out the right ways how to make all these things work. What's the right information flow between all the different nodes in order to get those nodes, which are typically an LLM call to work? Do you want to do few shot prompting? Do you want to fine tune models? Do you want to just work on improving the instructions and the prompt?

7:05And so I think there's a lot of how do you test those nodes? That's a big thing as well. How do you get confidence in your LLM systems and LLM agents? And so I think there's a lot of workflow around that to kind of like be discovered and figured out. One thing that's sort of come up repeatedly relative to agents has just been like memory. And so I wasn't sure how you think about memory and implementing that and what that should look like. Because it seems like there's a few different notions that people have been putting forward. And I think it's super interesting. So I was just curious if I were thinking on that.

7:35I also think it's super interesting. I have a few thoughts here. So I think there's maybe two types of memory and they're related, but I'll draw some distinction between kind of like system level procedural memory and then like personalization type memory. So system level memory, I mean, more like what's the right way to use a tool? What's the right way to accomplish this objective independent of who exactly the person is and how I'm different than Sarah or something like that. And then for the personalization bit, I think it's like, okay, you know, Harrison likes soccer and he likes basketball.

8:12And I should remember that when he asked questions. And so I think there's maybe slightly different ways that we see teams thinking about both of these. So on the procedural side, I think the main thing that we see people doing and that we think is pretty effective is a few shot prompting and maybe fine tuning for how to use tools, because that's basically what it comes down to. What's the right way to use tools? What's the right way to plan? And we see few-shot examples being really, really impactful for that. And so that's something where, and so I think there's this really interesting data flywheel of monitoring your application, gathering good examples, and then plugging those back into your application in the form of few-shot examples that we're pushing really heavily with Lang Smith right now.

8:57And then the other side of it is this like personalization level memory. And I think there's a few different ways to do this. Like I think OpenAI implemented it in their chat, in their chat GPT, where in the way I think it does it under the hood is it basically has functions that it can call to say like, remember this fact, or delete this fact. And so that's a really interesting, like active, active loop that the agent is engaging in where it explicitly decides what it wants to remember what it doesn't want to remember. I also think one thing that I'm bullish on is a more kind of like passive background process that kind of looks at conversations and almost like extracts insights.

9:36And then you can use those insights in kind of like future conversations. And I think there's pros and cons to each. And I think it speaks to the memory in general, I feel, is like a field that's just like super, super nascent. Like I actually am underwhelmed at the amount of like really interesting stuff that's going on there. And so I think a bunch of different approaches, no kind of like overwhelmingly best solution. Has the sophistication shape type of application that you see people building with Langchain or just generally in the ecosystem dramatically changed over the last few months? I do think that there are more examples, kind of as Elad mentioned, of agentic applications that are much more productive and more sophisticated, like multi-step rag systems with much more useful ranking.

10:24Does that match with the patterns you're seeing? Or what are you seeing that excites you the most that you think is most useful? That does generally match. I think LangChain from the beginning has always been focused on those types of applications. and not only the open source, but also Lang Smith, the platform. So I think, you know, a lot of the emphasis that we put into like the testing and the observability is really focused on these like multi-step things. We've always been focused on those. Probably it's generally true in the market that there's been more of a trend towards those. But from our perspective, we've always been focused on those.

10:56And so I think, you know, that hasn't been as dramatic. I think there has been like interesting things within that that have emerged. Just calling out a few things. Within RAG, I think we've seen really interesting and advanced query analysis start to come into play. So you're not just passing the user question directly to an embedding model. You're maybe doing some analysis on it to figure out which retriever should I send it to or what is the bit that I should search? Is there kind of like explicit metadata filter? So some and then so now retrieval is like a multi-step process and more there.

11:29And it's specifically around query analysis, few shot prompting and that whole data flywheel. I think we're starting to see come into play more on the agent side. I kind of alluded to this earlier, but I think, you know, the way that we've kind of thought about things is there's kind of like chains, which are sequential steps. You're going to do this and then you're going to do this and then you're going to do this and you're always going to do those in the exact sequence. And then, you know, last March or April or whenever, AutoGPT came out. And it was like, we're literally just going to run this in a for loop and it's going to be, you know, this autonomous agent.

12:01And I think the things that we see making it into production and informed a lot of the development of Lange Graph is something in the middle where it's like this controlled state machine type thing. And so we've seen a lot of that come out recently. And so I'd maybe call out that as like one thing that we've really updated a lot of our beliefs on over the past few months. Yeah, I think a combination of that and tree search and just like trying to be efficient with like your sampling at every step has shown like a lot of really interesting, effective applications recently. And I think the like cognition as one example of like a surprisingly amazing agent has come out.

12:43Like, where else do you think agentic applications will begin to work or that you've already seen? I think on the customer support side, that's a pretty obvious use case. I think Sierra, you know, has emerged there and is doing quite well there. I think, yeah, the Cognition demo was very impressive. I think they did a lot of things right. I think they really nailed a really interesting UX. And that was maybe one of the things that I was most excited about. And then obviously, it seems to work very well. And so I don't know exactly what they're doing under the hood. but those type coding problems in general we see a lot of people working on I think there's a really nice feedback loop that you can get by just executing the code and seeing if it works as well as the fact that people building it are developers and so they can test it coding, customer support there's some interesting stuff around

13:38recommendation chatbots almost. So I draw a distinction between that and customer support, or with customer support, you're maybe trying to explicitly kind of like resolve a ticket or something like that. And the recommendation bit is a bit more focused on like a user's preferences and what they like. And I think we've seen a few things emerge there. But I'd say customer support and coding are the two. Klarna as well, you know, they came out and had a pretty good release. One pattern that I think is very popular, and I can't tell if it is real or transient, is whether or not companies will be able to switch between different LLM models, right?

14:21whether it's a self-hosted, dedicated inference instance for them or if it's an actual API provider. But for any given application, take your prompts and go from Anthropic to Mistral to OpenAI to something else. In reality, it feels like the way an application responds is probably going to be sensitive to the fact that these LLMs are actually going to predict differently? Like, what do you think about this? Can you switch? Is that a real pattern? It's not as easy as it seems like it should be. And I think the main thing is that the prompts still need to be different for each model. I do think the prompts will probably start to converge in the sense that if you think the models are getting more and more intelligent, then like hopefully these small idiosyncraticies don't matter as much.

15:18And as more and more model providers start supporting the same things, then that will make it easier. And what I mean by that is, you know, so many prompts for OpenAI, which is, you know, the leading and most used one, use function calling. And, you know, up until some period ago, like no other models did. And so you just like couldn't use those prompts at all. But now like Mistral has function calling and Google has function calling. And so I think they're a little bit more transferable there. What else is on that list? There's function calling, there's visual input, like what else is going to differentiate these model APIs.

15:51Context Windows one as well. So I think this gets to like, yeah, what's the right context that you can be passing if it's longer? You know, if that changes, then that changes, that doesn't, like that changes the whole architecture of your application. Modalities one. Prompt injection for safety. Yeah, I think that's interesting. I think that's a real enterprise concern. And I think a lot of the agents are still just figuring out how to make agents work. This is a different axis almost. But to the point around like switching models, I do think we see a desire for this, especially when you start going to scale.

16:27So I think it's like make something work with GPT-4, but then, OK, you're rolling it out. Are you really is that like, you know, are you really going to eat that much cost with GPT-4? Can you use GPT-3.5? Do you want to fine tune? And so I think that transition is where we really start to see people thinking about switching models. There's definitely some switching models at the beginning, like if you just want to play around with different models and see their capabilities. But I think the most pressing need to switch models happens when you go from prototype to scale. Cost and latency would be differentiators there as well.

17:02One thing you mentioned I thought was really interesting is just context windows. And obviously Gemini launched with a million token context window. And I was just curious how you think about context window versus RAG versus other aspects of the model and how all those things tie together. And, you know, once we get to very long context windows and the tens of millions of tokens, like, does that really shift things radically or how does that change functionality? And so I was just curious, since you've thought about how all these things piece together, I was just curious how you think about those different factors and what they mean.

17:33Very good question that a lot of people are thinking about who are a lot smarter than me. I think, I mean, a few thoughts. I think like longer context windows definitely make like single shot things much more realistic. Like extraction of elements in a long PDF, you can do that one shot. Rag over a single long PDF or like five long PDFs. Okay, cool. You can do that. You can do that one shot. There are, I think there are definitely things at scale that don't fit into a single context window. There are also things where it requires iterations. You need to decide what to do, interact with the environment, get that back.

18:13So this whole idea of chaining and agents, that's less around context windows and more around interacting with the environment and getting feedback. And so I don't think that's going anywhere. I think with respects to RAG in particular, because I think that's where it often comes up, like, did this kill RAG? I think there's a few things. Actually, just today, one of our team members, Lance Martin, there's that like everyone's doing the needle in the haystack thing. And now all these models are like green across the board for whatever reason, they've all figured it out. But I think like that, that actually really doesn't reflect a lot of RAG use cases, in my opinion, because like, that's the needle in the haystack is like, okay, given this long context, can I find a single information point?

18:53But oftentimes RAG is about seeing multiple information points and then reasoning over them. And so I think with the benchmark he released is exactly that. Like as you increase the number of needles, you know, performance goes down, as you might expect. And then also when you ask it to reason rather than just retrieve, the performance drops as well. And so I think there's more work to be done there. And then I think another thing is just around the ingestion for RAG in the indexing process. Like a lot of attention has been paid to like text splitting and chunking and all of that. And I don't know exactly how that will change.

19:29Like, will you still do that, but you now just retrieve the whole document? Like we have a concept in Langchain of like a parent document retriever, which basically creates multiple vectors for each document. So maybe you just do that. Maybe you still, maybe you chunk it up into larger chunks and just retrieve those larger chunks. Maybe use a traditional search engine, like Elasticsearch or something. I'm not sure. That's probably the place I have the least confidence in. The one other area that I see a lot of people talking about and I see a few people actually doing is fine tuning. And to some extent, I think that's because with fine tunes, you lose generalizability.

20:03And so people just start focusing on prompt engineering or other ways to effectively get the same performance without the actual fine tune. But it's something that feels very awkward on and people talk about it a lot and people talk about doing it a lot. You probably have a great perspective since you see so many different types of customers. Are you seeing a lot of fine tuning happening in the wild? And if so, there's specific common applications or use cases for it. We see people experimenting with it. I think that only real place where they're doing it is when they've reached like really critical scale, which I still don't think is that many applications to date.

20:34I think there's a lot of difficulties with it. One's like gathering the data set for it. And so I think a lot of the things we have in Lang Smith tackle a lot of these issues, but like gathering the data set for it. So like having that data visibility and starting to curate that data set, evaluating the fine-tuned model. So like evaluation and testing is a huge pain point there that we're trying to tackle in a few ways. The third is just like, yeah, back to this point of people are still just like experimenting so rapidly. It's much harder to change a fine-tuned model than it is to change a prompt or even change few-shot examples.

21:08And so I think we're seeing more and more people use few-shot examples, but not a ton graduating to the fine-tuning just because, yeah, I think much harder to just like iterate quickly on. In terms of other major changes in the landscape, it's been it's been a big year. The first commit to Langchain, I think, was in October of 22, which is like when I launched Conviction as a fund as well. At that time, we didn't have Lama 2. We didn't have Mistral. There were not nearly as many open source models with what people would consider to be more useful reasoning ability. Has that changed in terms of like what you see application developers do with link chain?

21:52Gemini too. Oh, and Gemini, yeah. Fun story about that. The original models that we launched with OpenAI actually deprecated like a month ago. So the like actual original link chain, you can't run because the models don't exist anymore. But yeah, like there's, I think we see increasingly interest in open source, but the reasoning abilities are still just like lagging behind cloud three or GPT four. And I think like for a lot of the applications that it kind of, it probably depends on the types of applications that you're building, but a lot of the applications that LinkedIn is focused on with this kind of like reasoning aspect, those are just so crucial.

22:34And I don't think we see super compelling. I don't, I still don't think we see super compelling reasoning abilities in the open source models. And maybe that's one of my hot takes. But I think for a lot of the LinkedIn apps, the open source models maybe don't live up to a lot of kind of like the Twitter hype or Twitter excitement, at least not yet. Zooming out, like you have a really broad view. Like, what do you feel like that no one is working on that's going to enable better applications that should be? I think the most exciting stuff is at the application and UX layer right now. I think that's where the most exciting stuff is there.

23:11One of the, I don't know if this is, this is maybe, this is more the capabilities side-ish, but like memory, I think is super interesting, especially like personalized long-term memory. I don't know if, I don't know if it's necessarily tooling so much that needs to be built there as it's just like an application in a UX that's really focused on that. And, you know, if I wasn't doing LinkedIn, if I was starting a company right now, I'd probably start something at the application layer and it would probably be something that really takes advantage of like long term memory. I guess at the high level, similarly, is there anything that you view as like a major prediction or things that will change over the next year that nobody's really paying as much attention to?

23:54Memory is a big interest of ours. And so I hope that we'll have some kind of like breakthroughs there. I think a lot of the specifically around learning from interactions, incorporating that back in at a user level in a similar vein. Also, this type of more like system level memory, I think, is really interesting and building up building towards this idea of almost like continual learning. So there's, you know, like, can you learn from your interactions? And you can do that in a variety of different ways. This may just be where we sit in the ecosystem, but one exciting and probably under-talked about way is this just idea of building up few-shot example data sets and really using those.

24:33I think it's much faster and cheaper than fine-tuning models. It's easier to do than trying to programmatically change the prompt in some way. That's still kind of like a bit of an art. Um, and so, yeah, continual, towards continual learning with few shot examples is, is maybe one like really interesting area that, that we're excited about. Can you help our listeners like, just imagine like a little bit more viscerally, like what, um, type of application experience that would enable like, you know, a consumer application or a business application of what that type of continuous learning would allow you to do?