AI Agents and Data Integration with GPT and LLaMa with Jerry Liu - #628

0:07All right, everyone, welcome to another episode of the TwiML AI podcast. I'm your host, Sam Charrington. And today I'm joined by Jerry Liu. Jerry is co-founder and CEO of Llama Index. Before we get into today's conversation, be sure to take a moment to head over to Spotify or your listening platform of choice. And if you enjoy the show, please leave us a five-star rating and review. Jerry, welcome to the podcast. Thanks, it's super great to be here. And thanks for having me, Sam. Yeah, I'm really looking forward to digging into this conversation. Of course, LLMs are on everyone's mind right now.

0:41And as someone who's keenly interested in platforms and enabling technologies, like the questions around all the infrastructure that we're going to need to build to adequately support LLMs, particularly in enterprise-type scenarios, is really interesting. And you founded one of the early open source projects in the space. So I'm looking forward to exploring that with you, as well as kind of your general take on the space and what you're seeing. But before we dive into that, I'd love to have you share a little bit about your background and how you got here. For sure. Yeah, happy to chat about this.

1:16So I started Llama Index back in November. Before that, I spent two years at Robust Intelligence, leading the monitoring team in ML, back when we were thinking a lot about the ops stack for the traditional machine learning pipelines and workflows. And then before that, I spent two and a half years as a research scientist at Uber. And then before that, I was a machine learning engineer at Quora, working on recommendation systems. So I've had a bit of experience in both industry research, as well as at startup. And fundamentally, that's kind of the intersection of my interests as well. And so it's cool to see all these things come together, especially with both excitement and LLMs, as well as the huge wave of new startups are popping up at every layer of the stack, from the tooling layer to the application layer to the foundation model layer to the data layer.

2:01And so I'm super excited to share some of my thoughts in this space. But I would say I've been interested in generative models for a while. I first discovered GANs back in the day, back in 2016, 2017, the things that generated like small pictures of bedrooms and animals and faces. That was actually probably one of the first things that got me interested in AI in the first place. I thought it was magic. I looked at the 64 by 64 pixels of a bedroom and I thought, wow, this is really cool. And I want to dive in a bit more into this. And that motivated some of my interest into getting a bit more deep into the research side of things as well.

2:36And so kind of been tracking language model progress for a few years now, but haven't really kind of thought about fully how to really interleave language models with the rest of the application ecosystem until relatively recently. And this was one of the main impetuses for Lama Index. Well, I'd love to have you start out by describing Llama Index and what you're hoping to achieve with it. I think I initially came across it, I guess, a little bit in the context that I mentioned. But specifically, I want to say early on, but this is really only a few weeks ago. Like, you know, there was ChatGPT and then everyone was like, OK, how do I create ChatGPT for my stuff?

3:14And one of the tools out there that seemed to be kind of positioning around that niche was Llama Index. in particular as a way to kind of tie language model to existing data stores. Tell me about how you kind of arrived at that as a direction and describe it all in your words. Exactly. Yeah. The central mission of Lama Index is to connect your data to language models and to really tap into all the capabilities of language models and utilize them on top of your own private sources of data. I can trace the origins of Lama Index. And we started with a bit more of a smaller scope, and very much more as kind of like a fun tool to play around with and to solve a more specific problem.

3:59So the background is that back in November, I was trying to build a sales bot, basically. And I started playing around with GPT-3, I thought it was really cool. And then I wanted to use it on my internal company data. And one of the first use cases I thought about was, we have a lot of these like customer conversations, I was part of some of our customer calls, how can I use this technology to basically synthesize a to-do list for me for the next meeting? Because otherwise, before every customer meeting, I have to spend 20, 30 minutes reviewing some of the notes from the previous call transcripts to basically see what's actually happening and what I should do next.

4:30And so I was trying to use GPT-3 to help with this. And then, of course, once I started doing that, that's when you start running into all the issues of actually trying to feed your own private data to a language model. And so the technique that people were kind of playing around with at the time, and I can describe this a bit more in detail was this idea of just like, you know, stuffing stuff into the prompt. I think we call it a few different terms. Now there's like prompting and context learning, all these different things, but fundamentally it's just conditioning. You're providing the right conditioning variable for your input and using that to send some sort of input prompt to the language model and get back a response.

5:03And so as I was playing around with how do I stuff this big call transcript into the prompt window at GPT-3, the prompt size is around 4 ,000 tokens, but the call transcript can be a lot bigger than that. And especially if you have multiple call transcripts across multiple customers. So then I thought, what's the best way to like structure this data in some way such that we can still utilize the prompt window of GPT-3, but almost use it as like a cache space and still maintain some sort of external RAM or hard drive or whatever analogy that makes sense in order to feed into this cache space so that you can still achieve the task, but make use of the kind of like limited prompt size that you have.

5:41So that motivated some of the initial design of probably some of the origins of GPT index, as it was called at the time, and now it's called Lama Index. It was this idea of just building some sort of general data structure in order to organize your data in some way that the language model could access and traverse later on in order to give you back the results that you want. And so it started off as this design project. We came up with this very initial prototype of like a tree-like data structure and then posted it and released it on Twitter. it got some initial traction and discussion, even if it didn't work super well at the time.

6:15And I think that's at a certain point, it got me, you know, like pretty interested. That's a good sign, I guess. Like if it doesn't work and people are excited about it. But like, it was clear that people were thinking about this. Like it's one of those things where the tree index, like there were certain limitations of GPT-3 being able to kind of like sequentially reason over data. And we can kind of talk about that as well. But it was clear that people were thinking about this idea of augmenting GPT-3 with a memory or some sort of like external storage system, right? So that it could still have access to this data, but this data would be structured in a way that GPT-3 could access.

6:50And so as I was working on this, there started becoming more user interest in the space. Then I really made a decision to basically try to convert this project, which was more of a design project at a time, into a set of useful tools and distractions that people could actually use to build applications on top of. And so that's where what GPT index and that Lama index has evolved into now, where it's basically a central toolkit for, again, connecting your language models to your data. Any individual and enterprise is going to have some private sources of data that they're going to want to tap into when they think about, you know, utilizing the capabilities of language models.

7:25There's going to be PDFs, PowerPoints, spreadsheets, all those types of files on your drive. There's also going to be, for instance, like workplace apps and a lot of data contained in Notion, Slack, Salesforce, et cetera. There's going to be data in your data lake, for instance, vector databases, structure databases, et cetera. How do you think about both what this data stack is going to look like, as well as tapping into all this private knowledge so that you could combine it and use it with power foundation models like GPT or LAMA or other types of models? So I think it leads to some pretty interesting technical challenges, which I'm happy to share as well.

7:59But I think fundamentally, there's really like two modes of inserting knowledge these days when you think about how do you augment a language model with your own private data? I was going to jump in and say, it strikes me that one of the fundamental challenges is that the language model is going to be the most useful if it's fully aware of everything that you have, everything that it has access to, all that data, but you can't give it all that data at once. You have to come up with creative ways to either pre-process and identify the data that it's going to need to solve a certain address, a prompt and be selective or what?

8:37I know the first has led a lot of people to, you know, tools like embeddings and vector databases and things like that. I'm curious kind of how you kind of contextualize that and over, you know, address that problem. 100%. So I'll talk about that in just a bit and taking a slight quick step back to like contextualize the overall space of like, how do you feed external data into a language model? Probably one of the first things that has evolved is just like, how do you perform gradient descent right on top of some data so that in some sense like gpt3 can't memorize data from your knowledge corpus and so if you just ask like chat gpt anything about anything that's public on wikipedia it'll generally know the answer and you don't really have to do any of this like retrieval augmented generation i think there's some issues and right now at least in kind of making some of this fine-tuning distillation accessible to the end user it tends to cost a decent amount you have to prepare your data in a certain way and you have to have some sense of how to experiment and evaluate with these training techniques in order to make it work well for the end user.

9:33And so I think as a result, there's this other paradigm we'll talk about now, which is basically just take a pre-trained model and then figure out how do you have the best retrieval model to augment the existing language model with. And so that's kind of the space that Lama Index operates in right now. And that's where people are starting to think about this intersection between vector databases, language models, and then how do you actually surface the best context for the given task that you want to solve. And so let me give like an initial example, right? So like for an existing question that you want to ask, let's say it's a very simple question.

10:07It's just like, what did the author of this book do during his time in college, right? And let's say you have access to the entire biography of the author. The way that you would typically make this interaction work with vector databases is that you would take this knowledge, store the document as chunks in a vector database and add an embedding for each chunk. and then you would pair that with a language model. So then during query time, when you actually ask this question to this overall system, we'll do top K retrieval by embedding similarity. We'll embed the query and then use it to look up, you know, the relevant chunks from the vector database and then take those chunks and then inject it into the prompt.

10:45And then the prompt will look something like, here's the context, put the context in, here's a question, now give me the answer. And so I think I would say a lot of people are discovering that basic paradigm these days where you're basically just doing kind of like general semantic search over your knowledge corpus in order to find the relevant context. And I think for relatively simple questions that require kind of like fact-based retrieval, that tends to work well, just because you're kind of embedding the semantic context of the query and just matching it to some basic semantic context in your knowledge corpus, and then feeding it roughly the relevant information that you need to generate a response.

11:17I think Lama Index sees ourselves as both doing that as well, right? Because like that is a pretty common paradigm. And in fact, that's probably one of the easiest ways to get started with Lama Index. But we also offer some of the more advanced primitives as well on top of this in order for users to really kind of not only be able to get back a response from relatively simple questions, but for more complicated questions as well. And so what's an example of a more complicated question that necessitates these advanced primitives and kind of talk us through some of those primitives those examples.

11:52100%. So one basic example, and this is not getting into more complex territory yet, is let's say you just want to ask a question that requires a summary over the document, as opposed to top K retrieval. For instance, your task that you want to send in is literally, give me a summary of this document, or how can you describe the overall tone of this essay? Now, the type of data structure that you might use in order to look up information for this request, probably should not use top K retrieval, because if you embed the semantic context of that question, you're going to get some random chunks from the document.

12:24Whereas really what you want is have a system of going through all the chunks in that document. And so one thing, just very basic example, is that if you're trying to build some sort of, you know, general query interface that can handle both fact based retrieval questions, as well as summarization questions, how can you first have almost some sort of decision maker at the top that can take in this question that you have, decide whether it requires summarization of your documents or fact-based retrieval, route it to the right sub data structure, and then rely on the kind of query interface by that data structure to give you back the answer that you want.

13:00So that's probably one of the most basic examples of just how do you actually have combined both summarization and semantic search in one query interface. As another example, let's say you have information in both your Notion database, as well as your Slack conversations. And you want to synthesize an answer explicitly from both of them. And so for instance, you want to propagate the request from a query down into maybe a vector database or some sort of index structure from your Notion documents. And you also want to propagate this query down to your Slack conversations. How can you route this query down to both data systems and then combine the information at the top level to synthesize the response to your question?

13:40That's another example. And if you generalize this a little bit, let's say you ask the agent to, hey, not only synthesize information, but actually give me like the trade-offs, give me the pros and cons between this approach, which is documented in this section versus this other approach, which is documented in this other section. And if each of those are stored in separate data systems, you're going to need almost some sort of higher level primitives that can decide, given this query, how can I route it to these different data systems, transform the question in order to fetch the information that I want from these data sources, and then give me this full comparison between these responses at the top level.

14:16And so I think the interaction between retrieval and language model synthesis, in my mind, at least, there is inherently some sort of coupling between the two. And there's going to be some repeated interactions as both between the language model and retrieval model, as well as with the data structures that you have and the nature of the data systems. And so we want to build a tooling to help users be able to create all these different customizable indexes and views of their data to allow them to actually execute these different types of queries. And going through those examples, you've got this information retrieval style use case or effect-based query.

14:53You've got the summarization example. It sounds like we're starting to identify a higher level of abstraction that different use cases will fall under. You've identified two. Is it more the case that there's some manageable number of these primitives, like 10, 20, whatever? Or is it that every use case is going to be a little bit different? And there are hundreds of or 1000s of kind of fundamental ways that people want to work with their documents. And so you need to just give them a very open capability. That's a really good question. I think there's two ways I think about this. One is that right now we see probably like a few of these different use cases, like five to 10 that people tend to want to get answers from over their data.

15:39And it's possible there's like a giant long tail of different tasks. And I think it really scales with the complexity of the task. Like if you kind of give a very vague description to this system, and you expect it to just logically break it down into different subcomponents and execute all these different subcomponents for you, then the complexity just scales kind of like exponentially with the number of steps that it requires to execute this task. We want to provide the basic customizable building blocks in order to allow users to set up the structure so that for any subset of tasks or queries that they want to run, they're able to get back the result that they want.

16:14And so both from the simple stuff like question answering to more multi-step stuff, like if you ask a multi-part question or you ask this kind of general question that really requires breaking this down into individual questions over your knowledge sources, we want to give users the tools that do that. I think now taking a step forward, the next natural evolution of this is the fact that you don't actually want the users having to specify like a slightly different parameter for every use case that they want to solve, like semantic search or fact based retrieval, you're going to have to specify some top K parameter.

16:46And then for summarization, you're going to have to specify like the set of documents you want to summary over. And then for kind of like more complex interactions, you're going to have to explicitly say, hey, I want to break this question down into sub questions, and then route these sub questions through the underlying data sources. I think the next natural step here is how can we actually automate this and unify everything under a single query interface, right? Because you don't want the user to actually spend the time to actually define all the parameters, define all the tools just to handle the subset of queries, when they're going to have to know a priori, they're going to have to know like a priori what these tools are going to have to be.

17:23And so I I think building this next level of automation that can just take in a query and then in some sort of agent style interaction, decide what's the most relevant tool for the job and how can I use this tool and almost like build this tool as well. That's something that we're super interested in, especially as it relates to data systems, something that has a deep view of your existing data system and can make the right decisions in order to give you back the retrieval results that you want. Whether it's picking these different types of index structures that we offer that can solve these different use cases, or from another data systems lens, being able to look at stuff like how do you optimize like token usage?

17:57How do you make this faster, right? Like given previous queries, how can I actually make this faster? How can I reduce costs for the user, all these decisions. And so we're super excited to kind of take some of these paradigms are emerging nowadays of task automation and apply that to this data system world as well. When you think about the interface between LLM-based data processing system and data sources of record, what does that interface evolve to look like? You know, for example, does it evolve to look like the chat GPT plugin model where, you know, we're going to teach our Oracle databases how to spit out natural language prompt responses, that kind of thing?

18:42Or do you think that there's some more efficient way of doing that? Or is that more efficient? What's your view of the way these things evolve? I think the way it's going to work is that there's actually a few different lenses of looking at this. And I think all of these are pretty interesting to think about. One way of looking at this is that the data stack is actually just going to get radically simplified over time, especially as language models take off. The amount of hand engineering you're going to have to take to go from raw data source to massaging that data into some format, to some structured format, and then doing ETL and loading into a database such that you can run some structured query like SQL over it.

19:17I think a lot of that stack is going to become simplified. And especially because we can trust language models as these automated decision makers. And so if you look at some of the most basic things that people are doing nowadays, LMs have a capability of just understanding raw text, even if there's just a bunch of stuff in there. So if you load in a PDF file, you just dump the entire raw text into the chat GPT prompt window and you get back a response. You don't need to write a bunch of parsers to like extract some stuff. Obviously some cleaning up of this data would be nice, but QBT can generally just understand what's going on in the context, even if it has a bunch of like HTML tags or other kind of like markdown tags in there as well.

19:55So there's a huge possibility for the overall data landscape to just become simplified as we can figure out how to leverage the capabilities of LLMs and these various components of the data landscape to just like simplify the number of steps it takes from raw data to insight for the user. That's one way of looking at it. Going back into the plugin discussion though, and this relates to some of the stuff I was talking about with this idea of kind of like automated reasoning over your data. I see Llama Index as a core module that you could plug into any agent. And I think fundamentally what's going to happen is that as you see this rise of language model agents, each agent is going to have some sort of things that it could probably specialize in.

20:37And so you're going to have, for instance, this like outer agent abstraction for your application that's deciding what is exactly the best tool for my job. And one of those tools could be Llama Index over some view of your data. And then within Llama Index itself, we're probably going to have hierarchical agents within Llama Index, each that's really good at trying to automate and decide what's the best decision and best index and what optimizer to use over your data. And so we could totally see that being a system where there's basically this like outer agent abstraction that's using us as a core plugin.

21:09They send in this input request to us as a service, and then we take in this request, and then we can have some sort of agent-based system over your data as well that can decide what's the best tool to use and give you back the response that you would want. So I think it's just very interesting to see this, like, you know, you're going to have these different networks of agents all kind of interacting with each other in order to give you back the task. Curious to have you elaborate on you mean slash think of when you say agent. I can kind of parse that in two ways. One is we're going to have kind of these layers of LLM type systems that are making decisions and acting on both the user request and the data to compose the ultimate system.

21:56And while yes, these things will get cheaper computationally and cost-wise over time. That sounds very complex and expensive in a sense and slow by today's standards. But then there's another view in which agent is just the cool word that we're using to say software or rules or decision framework or something like that. And you're just talking about software. What is it that makes something an agent in your mind like? It's a good question. I think the definition of agent is going to evolve over time for what it's worth. And I have an initial view of that. And I think this view will absolutely evolve over time.

22:32I see agents as a layer of automation for decision making over any sort of function that you want to run. So for instance, you could, for instance, in your code, write a bunch of if statements that could decide, hey, you know, if this condition is true, do this function. If this other condition is true, do this function. I see an agent as basically taking in some input and doing that reasoning under the hood to decide, make a decision over some input, as well as some access to some context, for instance, over your data or over the set of tools that it's able to have access to. And it's able to decide what's kind of like the best next step to take given the previous step, the context, as well as the set of tools.

23:12And so I think fundamentally, to me, at least right now, even that on its own, which will probably evolve over time, is a pretty powerful paradigm because it just means that it's adding more layers of automation to the end user because now the end user does not have to set up all these structures on their own. They can set up stuff as like functions. And then in terms of routing between functions, they can trust this agent-like abstraction to make that decision for them. And so there's a lot of stuff around some of the core toolkits of LLAMA index that I think could be super valuable when coupled with this agent abstraction.

23:46Now, having a chat about that in just a bit, I think another thing you mentioned, though, was this idea of like cost and latency, which to be fair, I think are really good points. I mean, I think one of the fundamental downsides of costs or of agencies is that the moment you chain a lot of these calls together, especially when you add a ton of context to the prompts, the cost tends to balloon upwards. There tends to just be a lot of latency. Every call to GPT starts or to OpenAI starts incurring two to three seconds of latency. And then the end user might have to wait a bunch of time before seeing any results.

24:18And so I think part of this is banking on the fact that cost and latency do go down, that you might start seeing kind of more fine-tuned distilled models that are a bit smaller that can make decisions a bit quicker. I think that's another thing that's super interesting. And then the other part is performance. I think the other aspect is that the more decisions you chain together, the more errors propagate over time, right? If you have like, you know, 95 % accuracy over a single step reasoning, then, you know, like the error rate, if you chain like three or four calls is going to be like, you're kind of like multiplying a bunch of these together and the error rate will increase the more calls you change.