The Enterprise LLM Landscape with Atul Deo - #640

0:08All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington, and today I'm joined by Atul Deo. Atul is General Manager of Amazon Bedrock. Before we get into today's conversation, be sure to hit that subscribe button wherever you're listening to today's show. Atul, welcome to the podcast. Thanks, Sam. Excited to be here today. I'm super excited to have you on the show, and I'm looking forward to kind of diving in with you and talking through kind of this crazy LLM landscape, generative AI landscape that is rapidly evolving. But before we do that, I'd love to have you share a little bit about your background and your role at Amazon.

0:49Sure. So I lead product and engineering for Amazon Bedrock. I've been with Amazon for the last eight and a half years. I started my career as a software developer many years ago. I wrote code for a few years, then felt the need to understand business concepts, went to business school and then worked for a few years in various functions such as corporate development. And then in the last few years, I've been busy building products at AWS. You know, one of the things that I've been kind of finding interesting about this point in time is that a lot of the kind of the more sophisticated enterprises had like, you know, the past several years, gotten onto the machine learning train, built up competency around traditional models and everything that came along with that, MLOps.

1:41And now there's still an opportunity to get value out of AI, but it's a totally different language that we're asking them to use and think about. We're throwing out terms like LoRa and parameter efficient tuning and RLHF. When I talk to folks, particularly folks that are at the executive level, they're overwhelmed with a lot of this language. And really, they, for better or for worse, mostly for worse, they want magic. I use this chat GPT thing. It seems like magic. I want that magic for my enterprise. Do you see similar things? I definitely hear a lot of this from customers. And I think customers want to understand or unravel what is really going on here.

2:26because for them, as you rightly said, they have some experience in machine learning at a smaller scale within their company, but this almost feels like a paradigm shift. And so let me just try to take a quick stab at demystifying some of that for the listeners, right? So in the past, customers or companies typically would embark upon building task-specific models for a particular use case. Let's just take a use case of detecting fraud in a particular organization or creating a chatbot for customer care. Now, what would typically happen in a large company is that particular leader would go to an SVP or a VP and say, hey, this is an important problem for our company and I need to build machine learning model for this particular problem.

3:13And I need to go hire a few scientists. I need to go annotate a bunch of data to build a model for this. And here's my bill. And the VP would evaluate the proposal on a case-by-case basis and would basically say, you know, the problem looks interesting. It's strategic. The cost is decent. Let's go build a machine learning model. And a few weeks back, the team came back and said, you know, we build a model. It's working great. And the VP is happy. Now, the problem is this approach worked well for a few use cases. And the moment customers or companies saw that the machine learning was effective for their organization, they wanted to scale it for kind of moving from tens of use cases to hundreds of use cases.

3:54within their company. And also they were finding problems with doing that because first of all, like machine learning experts in this world are still limited. It's 2023, but the problem which was true in 2013 is still true in 2023. There are only so many data scientists and ML experts around out there. And furthermore, the cost and the effort of doing custom annotation can be pretty significant. Like it's just a lot of effort, you know, and it's not just about finding the humans to do it. you need the data scientists and the ML experts to again be involved in kind of defining what annotations need to happen.

4:28They need to kind of evaluate the quality of the annotations that come back and essentially ensure that a machine learning model is getting the right quality of data. Now, this process can be pretty onerous for a lot of small and medium-sized businesses, let alone large companies, right? So I think with foundation models, the big thing that has changed is that you're no longer training individual task-specific models. There's almost like this centralized training of one large model that makes use of a large amount of unlabeled data. Now, I'm stressing on the word unlabeled data because unlabeled data is easy to get, right?

5:05Unlike labeled data where it depends upon a lot of human effort, you can basically get a lot of data from the internet, from a bunch of different proprietary sources, and you can use them at scale without having to depend upon an army of people to go label the data, basically saying a dog is a dog and a cat is a cat. Simplifying the labeling process, but doing that at scale, even for example, an image, right? Identifying every single part in a single image can be pretty onerous. So doing that has really freed up the ability for people to train these large language models. Now, what happens under the code is - You're almost making it sound easy though.

5:43It is. Actually, in some ways, the bottleneck or the challenge is, you know, you have to collect a large amount of unlabeled data. You do have to apply hygiene on kind of using this data for training. Then there is you need a large amount of compute to make use of this data. And now the way it happens is you throw all this compute and the model makes use of algorithms to learn the relationships between the words on its own. You are not really kind of teaching it anything in terms of label data. The model understands sequential relationship between words and then says, you know, I understand basically that, hey, if there are a set of words before this, I roughly know what is the next word that's going to follow based on, again, I'm simplifying for our illustration purposes.

6:27But at the heart of it, that's basically what is happening. And every time the model generates next kind of output, it uses that as the input to predict the next set of outputs. So it's basically called as autoregressive behavior. Now, at the end of all this, what you get is a well-trained model, which has completed its pre-training process. And it is capable of dealing with a large number of tasks out of the box. Now, the standard set of tasks for a generative text model could be things like sentiment analysis, classification, information extraction, being able to converse. Again, the list goes on and on.

7:04There's a lot of tasks that come kind of automatically as part of the pre-training process. and now customers can take this model and adapt it for their particular use case within a company by just using a very small amount of label data. Now, this dramatically simplifies some of the things that we discussed earlier, which is companies had to go hire teams, invest in a lot of compute for training individual models, and now they just get a ready-made pre-trained model at somebody else. In this case, Amazon, we do pre-train our own models as part of Bedrock, And also we have partners like Anthropic who have developed these large foundation models.

7:41But somebody else has done the work, the hard work for them, which is a great starting point, right? That's kind of the way to think about it. Companies get an incredible starting point that they can take and either use it out of the box. In most cases, they can use it out of the box. In some cases, they may have to customize it or there are different ways of customization, fine-tuning, continued pre-training, reinforcement learning with human feedback. So there's some different, and LoRa, like the parameter efficient fine tuning. So that's kind of the general kind of structure, which is use it out of the box or customize it by changing the model's weights.

8:14Yeah, I kind of quipped that, hey, you make it sound easy. And certainly in many ways, it's easier than, certainly using a pre-trained model is easier than if you need to train it yourself. But there are still challenges associated with getting these models to do the productive things that you want them to do. I'm curious, what are the things that you find folks running into when they're trying to put these models to use? Absolutely. So that is a key question because think about the analogy. I'm going to start off with an analogy, and then we will get into the exact specifics. So think about a company that hires an extremely smart person from the outside, takes them, hires them, but puts them in a conference room with no access to their internal systems or internal documents.

9:12And this employee is not going to be very productive. So a raw pre-trained LLM is almost like this employee who is sitting in a conference room with no access. So what needs to happen is this LLM or this employee in this analogy needs to be given access to different tools and it needs to be made cognizant of the different data sources in the company. That's when the LLM can be truly productive for the organization. And there are different ways of going about it. Now, one approach is to use the model in an out-of-the-box capacity and then the other approach is actually customizing that using the proprietary data.

9:50Now, let's dive deep into each of those different kind of approaches. Now, in the first one where you can use the model an out-of-the-box capacity. There are, you know, again, for most listeners of your podcast, they may be familiar with the term called as prompt. That is the basic interaction model with which you communicate with large language models. Essentially, it's a natural language command. The simplest example is, hey, here's a doc, can you summarize it? And boom, you basically pass a doc and you get a summary as a result. And this simple way is called as zero-shot learning. Essentially, you give a command, you get a response.

10:25Now, the next kind of, kind of on the spectrum, the more complicated one is something called as in-context learning, where you say, hey, I'm still dealing with out-of-the-box. I'm not changing the model suites in terms of customization, but I'm basically now saying, here are a few examples of how I want you to do. Just do it this way. And this is all part of the prompt. The model takes a look and basically says, I got it. Here's the answer. So that's in-context learning. Or few-shot learning. Or few-shot learning. Correct. Now, in-context learning, it's basically a very powerful tool because there are a lot of models out there that essentially have this concept of context length.

11:03How much data can you pass as part of your context window that allows you to do more and more powerful things? Now, some of the models from Anthropic, for example, they recently announced that they are supporting 100 ,000 tokens as the context length. Now, that is pretty compelling, which means that you can pass really long documents, really long set of examples, or you can just pass a lot of contextual metadata as part of the prompt and then get your answers without actually doing customization by tinkering with the model weights. Now, let me be more specific. Now, there is a technique called retrieval augmented generation that has emerged in the last few weeks.

11:43Essentially, this goes back to the point of giving a smart employee access to your company's documents. Now, in some cases, I may want to get the model to actually do Q &A on my documents. Now, the way to do that is I can pass some of the relevant pieces of the documents in my company as part of the prompt itself. And basically then ask the model, hey, here are a bunch of documents. Here's my question. Can you go take a look at these documents and think through them and basically give me a coherent response? And now with essentially within context learning or kind of RAG, essentially you get precisely that.

12:22RAG is retrieval augmented generation? Correct. I hear this as probably the number one kind of generative AI use case or request. My experience in it is usually voiced as, hey, I want chat GPT, but for my data. Yeah. So Sam, the way companies do that typically is that, let's just take the use case of being able to answer questions on top of company's knowledge base. Now, to do this, there are several steps involved today. And obviously, these steps are going to simplify over the next few weeks or months as more and more companies understand the customer pain points and simplify them. So the first step today is a company has a bunch of documents in their knowledge base.

13:06They essentially take these documents and then they chunk them into smaller pieces. So chunking is essentially converting them into small paragraphs or lines, relevant portions. And then essentially you pass those chunks into what we call as an embeddings model. Now, an embeddings model is a type of large language model that takes a text as input and returns embeddings or mathematical representations that capture the semantic meaning of the input text. Now, once you get these embeddings, you put them into a vector database. There are many popular vector databases out there, such as Pineco and VV8 and OpenSearch.

13:48Now, once a company has gone through this process where they have converted all their internal documents, they've chunked them, they've created the embeddings, and they've stored them into a vector database, let's just take a hypothetical user query. Like, hey, I want to find out a response to a question, which is, what is the vacation policy for my company for a full-time employee? Now, behind the scenes, what's going to happen is that user query is going to be passed into an embeddings model, and you're going to get embeddings associated with that input query. And then under the hood, you basically do some matching between the embeddings for the user query and all the embeddings stored in your vector database.

14:30and essentially the relevant chunks of the text documents are extracted and then they are passed to the large language models as part of the context. And now when the large language model sees the original question, which is the query from the user, along with the necessary or the relevant pieces of documents, the large language model can then essentially formulate a very human-like answer that answers the precise question. Now, again, these are all steps happening in the background. And what the user just sees is a well-crafted answer that sounds very human-like. And it is a huge improvement over some of the traditional approaches that have existed in the past, which made use of things like keyword-based search.

15:18So essentially, with the large language models, both the generative model and using the combination with the embeddings model, Essentially, we are reinventing a use case like semantic search. Now, just while we are on the topic of embeddings model, this model actually, which doesn't get a lot of love out there on Twitter and whatnot, but compared to the main generative model, but it's a very nifty model. It is used for many interesting use cases like clustering, anomaly detection, and things like that. But semantic search is really one use case where it shines and it's kind of made some headlines.

15:52When you describe this process, it sounds very straightforward. What are you seeing in terms of the challenges that folks run into trying to do this at scale? Everybody wants this thing, right? They all want this, but not everyone has it yet. It must be harder than, hey, let's just throw a pine cone and suck all our documents in and we'll have this capability. What are the big sticky points? So I think there are a couple of things. I think companies are still trying to figure out which models they use, which providers they use. And they also want to figure out whether they are going to build some of these applications in-house or whether they want to ready to use application themselves.

16:37And also, I think overall, I do think that companies don't necessarily want to think in terms of vector databases. They want to basically think in terms of, here are my documents. And here's the solution to answer questions on top of my documents. They don't want to really deal with all the nitty gritties of this workflow that I described. I think on a related topic, which is we gave the example of that well-educated employee that's hired without having access to tools and data sources. So we touched upon, as part of this retrieval augmented generation workflow, we touched upon a lot of for the data sources.

17:14But there is also an interesting kind of trend evolving in the last few months, which is based on this paper called React, which is reasoning and action, which I think is one of the, I don't know if it got as much attention, but it did. I mean, obviously, people have been using the concept behind it like crazy. But essentially, what it does, it allows companies to make use of the LLM, invoke APIs, and it relies a lot upon the model's reasoning capabilities. So the way to think about this is... This is essentially the idea behind tools or plugins that an LLM can use to take action in the context of the response.

17:54Now, just in terms of setup, what happens typically is some administrator will have to create an agent and basically say, this agent's purpose is to do so and so. And it has to specify a list of resources. The resources could be a different set of APIs and what an API is supposed to do, what are the different parameters, and just plain English descriptions of what those parameters are supposed to do. And what kind of outputs does that function return or the API return, right? And then things like that. And similarly, they can also add some data sources. Now, what happens behind the scenes is ultimately you're still doing, at the end of it, you're doing a retrieval augmented generation.

18:31Because you're passing all that metadata as part of the prompt. And the model then kind of based on the user query, it figures out which of the APIs and which data sources should I use. It can make multiple invocations of the model. And the way it works is I may go to a database, I may get some results, and then I pass that to the model. And then the model says, let me think about this. I actually need something else. Then it says, okay, I need to get information from another database. So it goes, gets information. Now the model is essentially creating, or the agent is essentially creating kind of this prompt on the fly that is helping the model think through the problem.

19:12And at the end of it, it's almost like the model has sufficient information to make the final kind of decision. and then it says, you know, it has, I have all the information I need and here's the final answer. Now, this process of having the model think on what information is presented and kind of doing it in a recursive manner, I think that is truly transformative. Also, this is the concept behind which you may have heard of the popular thing, which is auto GPT, where essentially you created this agent and then the agent kind of was able to quickly kind of understand a broader goal and then kind of create subtasks and kind of keep going and solving them on its own.

19:48And particularly when you are able to combine things like do web search as one of the tools in its arsenal, I think there are a lot of interesting kind of avenues that open up. But I just wanted to kind of talk about this in the context of retrieval augmented generation and what are all the cool things that one can do simply by kind of not even actually customizing the model by tinkering with its model weights. Now, the other kind of side of the story is that in context learning, you need larger models which are capable with long context lengths, and you just do it on the fly without any fine tuning.

20:25But on the other end of the spectrum, what we hear from companies is, I have specific set of use cases, and I am very cost sensitive. I would rather take a more kind of appropriate model which gives me the right combination of accuracy, latency, and cost. And I can just fine-tune that for my particular use case using my proprietary data. Now, that approach is also a very fine approach because you're not then constantly dealing with a large model, and particularly in situations where latency is a factor, where latency and cost are factors, you may just want to go down that route with fine-tuning.

21:02Again, in fine-tuning as well, There is the full-blown fine-tuning, and then there are things like a lightweight kind of parameter-efficient fine-tuning and things like LoRa that have emerged in the last few months. You mentioned cost in there. Are you suggesting that the fine-tuning approach is more cost-effective than in-context learning? So here's the thing, right? If you want to do pure in-context learning, two things have to happen. One is you need a model with the longer context lengths, because let's just take that example of kind of auto GPT, where you are going to probably pass a lot of kind of API expects and recursively, it's going to keep adding context.

21:40You need probably long context lengths. The second thing is you need is a well-trained model, which can be large in size and typically larger in size means that the hosting can be expensive because it needs more expensive machines under the hood to do inference. So because of these two factors, again, they tend to make inference more costlier. The longer context length also adds to the hardware requirements in terms of hosting. So as a result, when someone wants to do that, those can be more expensive compared to when I just want to do fine-tuning for a particular use case or using a smaller model compared to the former.

22:20Okay. The challenge that I think this is, again, there is no kind of, we are still in, the jury is out. I think for some use cases, I think we are seeing where customers are just going to rely upon this paradigm of larger models in context learning. And in some use cases where customers are kind of sensitive about certain kind of requirements, particularly around latency and cost, they might just skew in terms of fine tuning. It's hard to tell right now which approach is the right one for a customer without actually understanding the specific details in their use case. So you've talked through a bunch of kind of these steps that are required to put these language models to good use.

22:59You've not talked much about the kind of production side of things. Like when we talk about, you know, traditional models, like there's this whole, yes, there's training and all that. but then there's all of the inference considerations and evaluation, and there's a whole set of considerations on that side of things. How much of that stuff do you see folks struggling with on the LLM side? You're absolutely right. There has been this entire ecosystem around MLOps. I think I do think that some of that ecosystem around MLOps is still relevant in this generative AI space or the foundation model world.

23:36Think about something like as companies fine-tune, different models. They're experimenting essentially, and they're experimenting to see which of these fine-tuned models ultimately I want to deploy in production. So they may want to actually do some experimentation and see kind of, for my use case, is this model, is model A delivering the right results or is it model C? And for that, particularly our service like Bedrock integrates deeply with SageMaker experiments and it allows customers to do precisely that kind of A-B testing and see kind of which of the ultimately the fine-tuned model they should deploy in production.

24:12And of course, there are many other things, things like pipelines, right? Where as a company, I may just want to fine-tune my model on a periodic basis as soon as I have X amount of new data. I may want to create some recurring things as part of my workflow. So I do think, and again, to answer kind of your question, I think on this, there are existing tools, but I also do think that we have to reinvent some of this tooling for the generative AI world. Because things like prompts, our historical way of dealing with systems was more structured. In the generative AI world, it's all in terms of kind of this prompt and it's kind of unstructured.

24:52While it's very powerful, it also creates a unique set of things because not everything can be classified like in a binary way as it's awesome or it's terrible. it always requires some human judgment in terms of kind of its output. Because let's just take an example of a summary. Model generated a summary. And if you ask two different people, what do they think? They may both say kind of, hey, the summary is correct, but I like one versus the other. And that's why I'm going to pick different one. It can be subjective. So I think we have to figure out kind of how to tackle that. I think also on the inference side, there are other challenges like hallucinations.

25:28A more common kind of thing with large language models is that while they can give a very compelling human-like response, they sometimes also tend to make up answers and completely inaccurate answers, right? Apparently, there was this lawyer who was trying to use a large language model and the large language model gave responses citing cases that were completely made up. Now, I think as a customer, as you are trying to rely on this, I think it's very important that customers are aware of these issues. And also as someone like us who is working on these technologies, we are taking various steps to ensure that we train our models in a way that they almost learn to say i don't know right so that is number one also techniques like retrieval augmented generation or rag they actually help steer kind of reduce the hallucination because essentially you're giving them context as part of the prompt and essentially guiding the model to say your scope of your answer should be mostly limited to this so stick to the stick to the context that's given to you to the script exactly exactly that's actually a good one i've never actually used it i might borrow this but again the challenge here is if you purely stick to the script then the distinction between the generative model and plain search disappears you cannot have the cake and eat it too almost so you still need some amount of creativity left in the model but at the same time you want to make sure that they are not kind of bsing so that's where kind of i I think the science will evolve in the next few weeks and months.

26:57Yeah. You mentioned Bedrock in the context of explaining some of the operational aspects of fielding LLM-based solutions. Talk about Bedrock. What is Bedrock? What role does it play? What are some of the key capabilities? Sure. So Bedrock is the easiest way for a developer in any company to build generative AI-based apps. It does not require this developer to understand any nuances of machine learning, and they don't even need to deal with any underlying infrastructure. It's a fully managed service that allows these developers to build any compelling apps using a range of different foundation models.