#160 Atul Deo: The Future of Generative AI at Amazon

0:00Bedrock is kind of the service that people come to to build their application. So, but it doesn't offer any UI support or whatnot. It's for educational purposes. But the core problem with large language models being used and certainly in any safety critical application, but in any application for which ground truth is important is hallucinations. Now we make our own chips here. Chips which gives us a tremendous advantage in these times where, you know, there is shortage of GPUs. Hi. Good tech solves problems you know about. Great tech solves problems you haven't even thought about. What can the commerce platform trusted by millions of merchants do for you?

0:44It's time for Shopify, the commerce platform revolutionizing millions of businesses worldwide. Whether you're a garage entrepreneur or IPO ready, Shopify is the only tool you need to start, run, and grow your business without the struggle. Shopify puts you in control of every sales channel, so whether you're selling satin sheets from Shopify's in-person point-of-sales system, or offering organic olive oil on Shopify's all-in-one e-commerce platform, you're covered. Shopify powers 10 % of all e-commerce in the United States, and Shopify's truly a global force, powering Allbirds, Rothy's, and Brooklyn, and millions of other entrepreneurs of every size across over 170 countries.

1:37Plus, Shopify's award-winning help is there to support your success every step of the way. Sign up for a$1 per month trial period at shopify.com slash IonAI. That's Shopify, S-H-O-P-I-F-Y dot com slash IonAI. That's E-Y-E-O-N-A-I, all run together. Go to Shopify.com slash IonAI to take your business to the next level today. Excuse me, sir. I couldn't help but overhear. Did you say Shopify? Oh, Shopify.com slash I on AI. Oh, carry on. Hi, my name's Craig Smith, and this is I on AI. I just got back from Las Vegas, where I was attending Amazon Web Services' re-invent conference, a cloud conference for developers.

2:43And I had the opportunity to speak with Atul Deo, who worked on developing Amazon Code Whisperer, a generative AI application to help write code, and now heads Bedrock, a service that gives developers a choice of high-performing foundation models to create generative AI applications. He talked about RAG, retrieval augmented generation, and building agents that execute tasks. Atul gives a clear and clear-eyed overview of today's tech landscape within the context of Amazon's ecosystem and beyond. I hope you find the conversation as interesting as I did. Hey, this is Atul Dave here. I'm the general manager for Amazon Bedrock.

3:34I manage, as part of my role, I manage product and engineering for Amazon Bedrock. Prior to AWS, my background was in computer science. I'm an engineer. I wrote code for a few years after an undergraduate degree. And then I basically realized I was just in a cubicle writing code without understanding the implications of what I was actually writing code for. And I realized I should probably get educated on some of the business aspects. Went to business school, went to USC for that. And then after business school, I joined Yahoo. This was back in 08, 09, when Yahoo was going through some interesting transition with the Microsoft acquisition offer and whatnot.

4:14And then I worked there for the next few years, and I transitioned from doing finance, business operations, and then moving into corporate development, where I focused on mergers and acquisitions for Yahoo towards the end. And then I moved to Amazon in 2014 and joined Amazon's corporate development team where I focused on mergers and acquisitions again for AWS. And as part of that role, I got to work deeply with some of our product leaders, product and engineering leaders in AWS. And I really, instead of just being on one side of the table where I am evaluating companies to acquire them, I wanted to kind of actually be a founder myself.

4:54And I wanted to do that in a very low-risk way. And I realized that AWS is probably the perfect kind of place to do that. And this was around 2016, 2017 time frame when machine learning, the team at AWS was being formed. It was kind of thriving under Swami. And I decided to join the team. And since then, I've been involved in a bunch of different projects. I helped launch Amazon Transcribe. Then I launched a number of AI-powered capabilities for Amazon Connect, which is our contact center software services. Specifically, they're infusing machine learning into customer care. So things like I launched a service called Contact Lens for Amazon Connect, which essentially helps transcribe customer conversations.

5:37I'm sure you've called a customer care scenario, like you called your airline company or your credit card company with some complaint. And these companies want to analyze kind of what conversations happen between their customers and their customer care professionals, right? And essentially, contact lens allows you to extract a bunch of different insights from those customer conversations. What are the sentiment? What are the key topics that customers reached out? What are the trends? Did the agent, the customer care agent, did they actually behave properly? Did they follow all the rules that a company had laid out for?

6:08And essentially, here we use the power of AI to do this analysis at scale and help the companies understand this better. I think from that point onwards, I was like, I think I had invested enough in infusing machine learning into other products within existing products within AWS. And then generative AI, the term was not really coined back then in 2020. but we had started to see rise of large language models and these models were materially different than what we had seen in the past. So I embarked upon development of Amazon Code Whisperer, which is our ML-powered code generation service. And I helped launch it in Preview.

6:49And then since then, I've been involved in development of Amazon Bedrock. So Code Whisperer is highly specific to code. But at that point, I basically realized that this thing is way beyond code. The generalizable nature of this technology could essentially allow people to build a host of different applications just outside of even code and helping developers kind of get productive. So that's basically the short deal. Yeah. So Bedrock is a service to help customers build generative AI applications. You were saying earlier that you guys don't really use the term platform. It's not there. There isn't necessarily an interface that you're working in.

7:33Before we get to Bedrock and Gen AI, I want to well, Code Whisper is generative. So I wanted to ask about where you are with the companies with Code Whisper. I'm not a coder. And two or three years ago, I was talking to people who were imagining the day when you could use natural language to talk to a computer and it would write code for you. So when Copilot and then Code Whisperer and the other code generation tools came out, I got very excited. when AutoGPT came out on GitHub, I got very excited. I thought, great, now I can talk to a machine and it can code something for me. And there was a plethora of articles in the tech press about, you know, I coded this app with CodeWhisperer or something.

8:37But when I tried to do it, it really is an auto-completion tool. It's not a code generation tool. How far are we from having end-to-end code generation tools that then you could pass the code to a review AI that would spot problems and fix them? So I would just kind of expand on that, right? So code autocomplete has been around for a long, long time. Like you had things like IntelliSense. I mean, I used to write code, as I said, in my early part of my career. I wrote a lot of.NET and C Sharp and Visual Basic code. And obviously, you do the famous kind of IntelliSense recommendation would give you kind of the next one word.

9:31Or kind of probably gave you, after a few years, it started giving you a little bit more. But that is materially different than what is happening now. So with these large language models, it's not just a tiny bit of autocomplete. At the very least, you're getting a quick one, completion of the entire line of code, or you're getting next several lines of code. And you can just even get, just say in plain English, that, hey, I want to write a function to do so and so. And based on the context of your prior code or rest of your code in your kind of project, you'd get next several lines of code. In fact, it'd complete the entire function for you.

10:05That is materially different than what we had in the past, which is just simple autocomplete of one or two words. So the key difference, though, in kind of where you want it to be versus where things are today, you still have to have the developer's mind where you need to decompose your broader problem into a set of smaller problems and then basically tell kind of the whatever tool you're using, let's say Code Whisperer, I want to write a function to do and so and so. But usually, as you know, real apps don't only have one function. It's usually a combination of many things. So the developer today still has to put together some sort of a plan as, hey, here's kind of my logical flow, and here are the various things that I need to do.

10:50So that's basically what is happening. That's the current state. Is it possible to use an LLM like GPT-4 to give you the steps to break down a problem into these sub-problems? So that's basically the right way to do it today, where you can use some of the reasoning capabilities of the model to basically say, I have a big problem, kind of how do I actually come up with the right architecture? And because the model has seen a lot of code in the past, it will generally tell you kind of, hey, what are the best practices? And if you are building a specific thing in mind, here are the kind of key kind of classes that you may have to create.

11:32Here are the key functions, key methods, and kind of here's the general structure. And it might even tell you here are the next set of functions, and it might start giving you code. But I think the key point, though, is ultimately the human in the loop still needs to roughly understand and verify. It's not that it's going to give you an app that's going to be perfect. It might do that for simpler apps, but inevitably, if you are a builder, nothing plain vanilla is ever acceptable to you. You want to put your own twist and turns on the thing that comes out of the model. I mean, that's always been the case.

12:05Builders want to be creative. Yeah. And the code review tools, if you generate an app that combines multiple functions that have been written by CodeWhisper based on the roadmap given to you by GPT-4 and it doesn't execute, are the code review tools strong enough to go through and say, well, you have an error here? So two parts. One is I'll tell you the inherent feature of Code Whisperer is it identifies security vulnerabilities in the generated code, and it is not just in the code that it generated because usually it's a combination of what a developer wrote and what was generated by the model, and it will give you kind of a list of security issues in the overall code base as in the current window.

13:07But separately, even code reviews, we already launched a service called CodeGuru, I think in 2019 already, which is even before the advent of generative AI, where it could analyze lines of code and just do code reviews using power of machine learning. Wow. Okay. And then, so I'll play around with that. But Bedrock, describe then what Bedrock is. Sure. So Bedrock is a simple API that allows any developer in a company to access some of the best large language models or foundation models without the developer having to understand nuances of machine learning or having to manage any underlying infrastructure.

13:50They don't need to know kind of what GPUs, what instance, none of that. All they need to know is what is the application that they are building. And they just need to be able to interact with the model in plain English, give them prompts, and the model gives you a response. Yeah, I was saying to you earlier at the expo part of reInvent, which is where we are in Las Vegas, there was a booth for something called Party Rock, which is built on Bedrock. and it's a simple user interface to build Gen.AI tools or applications. The guy was showing me that you can pick your LLM. So to me it looked a little bit like an orchestration layer where you're picking your LLM and then you're talking to that LLM.

14:53Is that right? That's exactly what happens in Bedrock. So as I said, the developer has a choice of different LLMs in Bedrock. We support a number of them from some of the best startups in the company in the world, and including Amazon ourselves. So we have developed a set of foundation models, which we call as Titan. It's a family of models. And developers can pick any of these models because, you know, different developers have different needs for different models based on their specific use cases. They may have certain requirements on accuracy, latency, cost, and whatnot for their use case. And depending on that, they will basically choose the right model.

15:28Right. And then in PartyRock's case, it has a UI that, you know, you put your prompt in, and then it builds a very basic app based on that. is one thing that I was asking and maybe you could answer. The problem with, and Party Rock is at this point kind of a sandbox for people to play in. It's not producing production. It's for educational purposes. Right. But the core problem with large language models being used in any, certainly in any safety critical application, but in any application for which ground truth is important is hallucinations. Is there a companion service within AWS? Yeah. So it's part actually we launched a feature within Bedrock.

16:40We call it Knowledge Basis for Amazon Connect. So essentially, as you may know, there is a very common technique in the generative AI world that has evolved over the last few months. It is called as Retrieval Augmented Generation or RAG. Right. So essentially, the large language models are pre-trained on a broad set of public data. And they don't really understand two things. They don't understand latest information because they were trained up to a certain point. So the second thing they don't understand is they don't understand the context of your company and your particular documents. So even within the company, the documents can sometimes be changing because maybe the CEO of a company decided to change the return policy.

17:23The return policy was certain on November 30th, and it is a new return policy on December 1st. And obviously, if a customer comes and asks the question about return policy on December 1st, you don't want to be giving it the wrong answer, right? So sometimes you want the latest and the greatest answer. But you want also at the same time the capabilities of the generative model to give you the right answer. So essentially, with retrieval augmented generation, you extract the right snippet of information from your company's data. And you include that as part of the context for the model when you pass your request.

17:58And the model takes into account two things. It takes your actual specific request. For example, I may say, summarize this doc. You give a doc to it. But here you may say, hey, here's a bunch of information. Here's the question. And here's the pertinent information. The customer is asking about return policy. And here's the exact updated snippet from our internal documentation, which talks about this return policy, which was changed yesterday. And here's the thing. Now the LLM is able to take that into consideration while it crafts its response for the end user. Right. And maybe you can talk a little bit about RAG because I haven't delved too deeply in it.

18:38Yeah, just describe the architecture, what kind of database you're using. Sure. So let me just describe RAG first in more kind of technical detail, and then I'll explain what our specific solution for that is. So essentially think about a company. So the company has a bunch of documents. Now, the first step they have to do is they have to break those documents into smaller chunks and then pass those individual chunks into what we call as an embeddings model. Now, this embeddings model is not really a generative model. It takes text as input and it generates mathematical representations or vectors for those chunks of text.

19:19Now, these mathematical representations capture semantic meaning of the text, input text in a way that it can understand, like the model can understand better. And then, essentially, the company has to, let's say it has a million documents. It takes this million documents and, say, creates 10 million chunks. And then takes those 10 million chunks, passes them through the embeddings model, and now it has corresponding embeddings. Now, these embeddings need to go in a specialized store, which is called as a vector database. Now, this is like a new class of related products that has emerged in the last few months.

19:58And this vector database is optimized for storing embeddings, but also retrieving embeddings. Now, let's say the query comes in from our example before. I'm trying to ask about a return policy. Now, I can't compare text with embeddings. I have to compare embeddings with embeddings. So when a query comes in asking about return policy, I again pass that same query to the same embeddings model, and it generates the corresponding embeddings for the input query. And now the vector database does matching between what it has in terms of its corpus, the embeddings corresponding to the 10 million chunks, and the embeddings corresponding to the input query.

20:44and then it's able to quickly find out, hey, based on this end user's query, I was able to find that right chunk which corresponds to the return policy. And it then returns me back the specific text. It says, you know, the new return policy as of December 1st is so and so. So now I was able to extract that. Now the next step is I've extracted this information. Now somebody has to do the work, particularly a developer has to do the work to then take this in real time, run time, and then pass, append it to the prompt. So the original question to the model or to the application was, what is the return policy?

21:23Now, the developer has to take this information from the vector database, append that to the, before it passed, makes that request to the large language model, and then the large language model generates the response based on that final compile. It's called, basically, that's why it's called as retrieval augmented, because you're augmenting that output from the vector database into the prompt, and that's why you're getting the generated response after augmenting it. Right. I mean, I've had on the podcast, Adonuf, who's from Datastacks, which is a vector database company, and I've had Aido Liberty from Pinecone.

22:05Yes. But they didn't use the term rag. is that because they're only building the database part? So now the interesting thing is vector databases have been around for a while, right? But they did not have this popularity with the generative aspect. RAG has this retrieval augmented generation. So that generation part only became popular in the last, I would say, 12 months since the advent of the popularity of large language models. So now let me just switch gears. So we've discussed about what RAG is. Now let me just talk about what Bedrock does for RAG. I think the answer is pretty obvious. There's a lot of undifferentiated heavy lifting here.

22:48The developer, as I said, has to develop, let's say the developer has a bunch of documents in S3. S3 is our place where people store their data. And now from that point onwards, they have to take these documents, chunk them, pass those chunks to an embeddings model, then store them in a vector database which means they have to go sign up for some vector database, create an account manage permissions and then they have to manage this runtime workflow of hey when a query comes in I have to then create the embeddings for that input query get the retrieved answer, append it to the prompt and get a final response there's too much going on so with knowledge bases a lot of these aspects are automatically managed but at the same time while they're managed, we give control to the developer.

23:36So the things like they get control on what kind of chunking strategy they have to use. There are different ways of chunking. I use a very simplistic example of saying 1 million documents, 10 million chunks. But there are different ways in which people can go about it. They can also pick a different embeddings model. They may want to pick a different vector database. So we are natively integrated with Pinecone, with Redis, in addition to our own vector database, which is the OpenSearch serverless engine, vector engine. So in fact, with OpenSearch, customers don't even have to go to the OpenSearch console.

24:15If they say they want to use OpenSearch, we actually create the resources on their behalf. And they get automatically, the embeddings get created for them automatically, and they can go and access it. And the vectorization of the documents that you want to put into the language model, does Amazon also have that end of the workflow where you just upload a million documents? Exactly, exactly. So the only thing a customer needs to do here is upload a bunch of documents in S3. After that, Knowledge Basis for Amazon Bedrock does all of this heavy lifting for you. And it also has a query API. So you can simply pass your end user query, which is, hey, what's the latest return policy?

25:08And all the sausage making in the back, all they get is the answer, based on the latest return policy documentation. And they get an answer. And then they can do many things with it. There's also a generate API, which they can just pass this to the specific model and then get the final response. So all that management is automatically taken care of for the developer. Yeah. Okay, I have another question that's kind of related. I've been talking to people a lot and hearing people complain a lot about rate limits, about the quota allowed an individual customer in terms of queries per minute or tokens per minute is not big enough to build heavy enterprise level applications.

26:03And I've been having kind of a debate with a CEO of a company who I know pretty well who says, oh, that problem is being solved by being able to compute inference on CPUs instead of GPUs by distillation, by using sparse models and that sort of thing. But then when I talk to people, other people, one guy in particular at a company called Nomad Data says, no, this problem is fundamental. It comes down to GPU availability, which comes down to silicon starts, which comes down to$20 billion fabs. You just need more fabs. So where's the truth? Yeah, so I would say a lot of what you see in terms of limits, this is a true problem.

27:05If you look at almost every service out there, people are bottlenecked on GPU availability. Like suddenly we had this massive spike. I mean, a few things happened at the same time. We had COVID, supply chain crisis, rise of large language models, massive demand for GPUs. And now not this time for crypto mining, but for LLMs, particularly in inference where people are seeing the value in what generative AI brings to the table. and everybody wants to infuse generative AI. But at the end of the day, there is a compute shortage. So that is basically what is happening. But one of the things that differentiates AWS here is years ago, in fact, this was exactly the time, almost 10 years ago when I joined Amazon, we had started investing in our custom silicon chip development efforts.

27:53And we acquired a company called Annapurna Labs. It's an Israeli-based... I'm sorry, what's the company? Annapurna Labs. So it's an Israeli startup. And while the initial focus was mostly around networking, because AWS, that's the core. We are good at infrastructure. So that was the initial focus. But quickly, the team started investing in specialized chips for machine learning, and specifically machine learning training and machine learning inference. And that gave birth to what we call today as Tranium, which is chip optimized for training, and Inferentia, which is chip optimized for inference.

28:32Now we make our own chips here. So of course we are still a great partner with NVIDIA, as you saw in Adam's keynote today. But we also have our own chips, which gives us a tremendous advantage in these times where there is a shortage of GPUs. So what I've heard from the people who are saying rate limits are a bottleneck that's here for a few years. They say that big companies are experimenting, building pilot products, but no one is putting major generative AI products into production on an enterprise scale because they just can't get the token throughput. Is that true? I don't think that is the case.

29:33I mean, we have plenty of customers who are starting to go production and do very interesting things. As I said, I think if the company, if they are using certain cloud providers where they are only solely dependent on GPUs, then that is probably going to be the case. But as I said, in our case, we are diversified here. where in addition to GPUs, we are also using a lot of tranium and inferentia. Yeah. What about GPT-4, which seems to be the preferred model for the time being? Does the constraint exist there? So, GPT-4 today, GPT-4 is not available on Bedrock. It is only available on Azure OpenAI.

30:16So, I'm assuming that Azure OpenAI is probably running on GPUs. Right, right. But so maybe this is a problem that exists primarily with GPT-4, but other foundation models, depending on how their architect or which ships are using. I think in your framing of question, you touched upon a lot of interesting topics. So things like quantization, things like distillation. And so I think instead of going into that specific nuances of that, I do want to say that, hey, you know, people have different problems and you don't need the most expensive, your most powerful tool, your most compute hungry tool to solve every problem.